Since XXXX, Software and Data Carpentry have collected information on learner demographics, perception of tools and confidence in working with data. As we continue in our goal to streamline processes as The Carpentries, the Assessment Team completed an analysis of the pre- and post-workshop surveys for both Software and Data Carpentry. The goal of this analysis is to understand the impact our workshops are having on learners, and how we can improve our surveys and assessment infrastructure. This report covers the workshops from NA to NA.
As an overview, 1495 learners have responded to Data Carpentry’s surveys, while 14185 have responded to Software Carpentry’s.
This report includes the following:
Learners attend Carpentries workshops for many reasons. Data Carpentry’s workshops are domain specific and focus on the fundamental data skills needed to conduct research. Data Carpentry’s Ecology and Social Sciences curricula begin with a lesson on data organization and includes data cleaning with OpenRefine. From there, learners spend time learning a base programming language, either Python or R.
Data Carpentry’s Genomics curriculum also includes programming, however the focus of this curriculum is best practices for organization of bioinformatics projects and data, use of command line utilities and tools to analyze sequence quality and perform variant calling, and connecting to and using cloud computing.
| Why learners attend Data Carpentry workshops | n | % |
|---|---|---|
| To learn skills that I can apply to my work in the future | 960 | 63.7 |
| To learn skills that will help me get a job | 445 | 29.5 |
| As a requirement for my program/current position | 103 | 6.8 |
XX% of Data Carpentry learners attend workshops to learn skills they can apply to their work in the future.
Software Carpentry workshops teach automation with the Unix shell, a tool that allows users to run commands interactively or by scripting. In Software Carpentry workshops, version control of source code with Git and GitHub are also taught for learners to learn facilitating contribution and collaboration on online repositories. Programming in R or Python is also taught. Software Carpentry’s curriculum teaches basic lab skills for scientific computing. Compared to Data Carpentry’s learners, Software Carpentry’s tend to have more experience witht the tools covered in the workshops, and learners come to learn new and/or additional topics (XX%).
| Why learners attend Software Carpentry Workshops | n | % |
|---|---|---|
| To cover new/additional topics | 458 | 74.7 |
| To network | 78 | 12.7 |
| To become a Software Carpentry helper/instructor | 39 | 6.4 |
| To help host/run a workshop | 38 | 6.2 |
The majority (XX%) of Data Carpentry’s respondents are either unsatisfied or feel neutral about being satisfied with their current data management practices. By data management practices, we include behaviors such as keeping your raw data raw, reusing code, and using databases, queries, and scripts to manage large datasets.
| Data Carpentry Learners satisfaction with current data management practices | n | % |
|---|---|---|
| Very unsatisfied | 100 | 6.7 |
| Unsatisfied | 408 | 27.3 |
| Neutral | 405 | 27.1 |
| Satisfied | 173 | 11.6 |
| Very satisfied | 22 | 1.5 |
| Not sure | 75 | 5.0 |
| Not applicable | 60 | 4.0 |
| Didn’t answer | 252 | 16.9 |
In terms of current programming usage, XX% of learners either never use programming, or use programming less than once per year, but no more than several times per year. Only XX% program on a daily basis. This is no surprise, as Data Carpentry workshops tend to attract novices.
| Data Carpentry Learners current programming usage | n | % |
|---|---|---|
| Never | 314 | 31.9 |
| Less than once per year | 151 | 15.4 |
| Several times per year | 168 | 17.1 |
| Monthly | 101 | 10.3 |
| Weekly | 131 | 13.3 |
| Daily | 118 | 12.0 |
Data Carpentry workshops are mainly populated by “word-of-mouth”. XX% of respondents learned about the workshop from either a friend, colleague, or their advisor or supervisor.
| How respondents find out about Data Carpentry workshops | n | % |
|---|---|---|
| My friend/colleague told me about it | 324 | 47.0 |
| My advisor/supervisor told me about it | 244 | 35.4 |
| Read about it in a newsletter or university web site | 85 | 12.3 |
| Other web site | 20 | 2.9 |
| Twitter or other social media | 17 | 2.5 |
On the contrary, the majority of Software Carpentry workshop respondents (XX%) find out about workshops through instituion mailing lists or flyers.
| How respondents find out about Software Carpentry workshops | n | % |
|---|---|---|
| Institution mailing list or flyer | 5152 | 75.3 |
| Conference/meeting/seminar | 641 | 9.4 |
| Funding organization or program officer | 357 | 5.2 |
| Our website | 357 | 5.2 |
| Social Media (Twitter, Facebook, etc.) | 294 | 4.3 |
| Journal or publication | 37 | 0.5 |
In summary, both Data and Software Carpentry workshop respondents attend workshops to learn about or improve upon their current data management and analysis skills.
As previously mentioned, Data Carpentry workshops are domain specific, and curricula include Ecology, Genomics, Geospatial, Social Sciences, and Reproducible Research. XX% of respondents learned R in their workshop, while XX% learned Python.
| Data Carpentry: Language Covered in Workshops | n | % |
|---|---|---|
| R | 808 | 64.7 |
| Python | 188 | 15.1 |
| I don’t know/I don’t remember | 218 | 17.5 |
| Neither | 35 | 2.8 |
The Carpentries is committed to making participation in our workshops a harassment-free experience for everyone, regardless of who you are, where you come from, or your experience with the tools we teach. We establish norms for interaction by having, discussing, and enforcing a Code of Conduct such that our workshops provide open and inclusive learning environments. XX% of Data Carpentry respondents either agree or strongly agree that they felt comfortable learning in their workshop environment, and XX% of Software Carpentry’s respondents agreed or strongly agreed the workshop atmosphere was welcoming.
Data Carpentry respondents were asked to rate their level of agreement with several statements regarding their instructor’s knowledge, instructional method, and enthusiasm. Their responses are in the figure below, and axis labels correspond to the statements as follows:
The largest impact we see is that XX% of respondents said they felt comfortable interacting with the instructors. We know that our instructors are the reason why our workshops are so well received. It’s also great to see that XX% and XX% of respondents felt our instructors were knowledgeable about the material being taught, and were enthusiastic about the workshop, respectively. We would like to explore what training would help our instructors so that the percentage of respondents who felt they were able to get clear answers to their questions from the instructors would increase.
Software Carpentry respondents were asked to rate how they felt instructors and helpers worked as a team based on the following criteria:
The two Likert plots below provide an analysis of respondent’s answers for both instructors and helpers.
From the figures above, we see that Software Carpentry instructors and helpers are considerate, enthusiastic, give clear answers to questions, and are good communicators. As a whole, our instructors work as a team and are successful in creating a warm and welcoming workshop environment.
One of the goals for Data Carpentry’s lessons is that learners are able to immediately apply what they learned at the workshop. The figure below shows that XX% either agree or strongly agree that they’re able to immediately apply what they learned.
As the majority of Software Carpentry learners attend workshops to learn new skills it is great to see that XX% of learners either learned mostly or all new information during the workshop.
We want to be proactive in ensuring learners have access to whatever they need to participate in a workshop. Both Data and Software Carpentry learners are asked to inform workshop organizers if there is anything they need that would make their workshop experience better. Data Carpentry’s respondents were asked if they had accessibility issues, and XX% reported they did. After reading the open-ended responses, we see that the issues were related to not being able to hear and/or see in the back of the room.
| Data Carpentry Respondents Having Accessibility Issues | n | % |
|---|---|---|
| No | 654 | 89.2 |
| Yes | 79 | 10.8 |
We use the Net Promoter Score to measure learners’ likelihood of recommending workshops to a friend or colleague. The scoring for this question is on a 0 to 100 scale. Respondents scoring from 0 to 64 are labeled Detractors, and are believed to be less likely to recommend a workshop. Those who respond with a score of 85 to 100 are called Promoters, and are considered likely to recommend a workshop. Respondents between 65 and 84 are labeled Passives, and their behavior falls in the middle of Promoters and Detractors.
| Data Carpentry Promoter Score | n | % |
|---|---|---|
| Detractor | 32 | 4.395604 |
| Passive | 131 | 17.994505 |
| Promoter | 565 | 77.609890 |
78% of Data Carpentry respondants are promoters (i.e. would recommend a workshop).
| Software Carpentry Promoter Score | n | % |
|---|---|---|
| Detractor | 32 | 4.395604 |
| Passive | 131 | 17.994505 |
| Promoter | 565 | 77.609890 |
For Software Carpentry respondents, NA% are promoters.
In summary, Data and Software Carpentry workshops provide a warm and welcoming environment whether learners are brand new to programming or have some experience. Attendees are recommending workshops to their friends and colleagues, and we know that our instructors and helpers are the major reason why.
Learners were asked to rate their level of agreement with the following statements related to Data Carpentry’s workshop goals and learning objectives. The figure below provides a visual representation of their responses, comparing them before the workshop and after the workshop. Axis labels and the corresponding question are as follows:
As the scoring for the above factors is ordinal from strongly disagree (1) to strongly agree (5), we show the mode (most frequent responses) for respondents’ before the workshop, and after the workshop. The comparison above is paired, meaning, we are comparing those who provided us with a unique identifier. This figure includes XX respondents.
In the figure below we show another representation of the pre- and post-comparison of respondents skills and perspectives.
As the scoring for the above factors is ordinal from strongly disagree (1) to strongly agree (5), we show the mode (most frequent responses) for respondents’ before the workshop, and after the workshop. The comparison above is paired, meaning, we are comparing those who provided us with a unique identifier. This figure includes XX respondents.
In the figure below we show another representation of the pre- and post-comparison of respondents skills and perspectives.
Software Carpentry Respondets were asked to tell us about their experience with these topics before the workshop:
The following is a comparison of Software Carpentry Respondents’ knowledge about the tools before compared to after the workshop.
Motivation is important, but being confident in your ability to complete specific computing tasks is an equally important goal of Software Carpentry. The grid below shows respondents’ self-reported ability to complete tasks including:
It also provides their self-reported level of confidence in being able to complete the tasks above after completing the workshop.
In Software Carpentry’s pre-workshop survey, respondents are asked whether or not their workshop takes place in the United States.
| Software Carpentry Workshops in US | n | % |
|---|---|---|
| Yes | 6489 | 45.7 |
| No | 4650 | 32.8 |
| Didn’t answer | 3046 | 21.5 |
| Data Carpentry’s Respondents by Discipline | n | % |
|---|---|---|
| Life Sciences | 444 | 22.0 |
| Agricultural or Environmental Sciences | 307 | 15.2 |
| Bioinformatics/Genomics | 292 | 14.5 |
| Biomedical/Health Sciences | 288 | 14.3 |
| Social Sciences | 122 | 6.0 |
| Mathematics or Statistics | 101 | 5.0 |
| Earth Sciences | 96 | 4.8 |
| Engineering | 91 | 4.5 |
| Computer Science | 88 | 4.4 |
| Business/Economics | 57 | 2.8 |
| Humanities | 53 | 2.6 |
| Physical Sciences | 53 | 2.6 |
| Library Sciences | 28 | 1.4 |
| Software Carpentry’s Respondents by Discipline | n | % |
|---|---|---|
| Life Science - Organismal/systems (ecology, botany, zoology, microbiology, neuroscience) | 2698 | 21.0 |
| Life Sciences (Genetics, genomics, bioinformatics ) | 2683 | 20.9 |
| Mathematics/statistics | 945 | 7.4 |
| Physics | 803 | 6.2 |
| Planetary sciences (geology, climatology, oceanography, etc.) | 787 | 6.1 |
| Civil, mechanical, chemical, or nuclear engineering | 696 | 5.4 |
| Medicine and/or Pharmacy | 688 | 5.4 |
| Social sciences | 591 | 4.6 |
| Chemistry | 578 | 4.5 |
| Economics/business | 483 | 3.8 |
| Psychology | 418 | 3.3 |
| Library and information science | 375 | 2.9 |
| High performance computing | 365 | 2.8 |
| Humanities | 319 | 2.5 |
| Education | 264 | 2.1 |
| Space sciences | 163 | 1.3 |
| Data Carpentry’s Respondents by Position | n | % |
|---|---|---|
| Graduate Student | 592 | 45.9 |
| Research Staff | 200 | 15.5 |
| Postdoctoral Researcher | 183 | 14.2 |
| Faculty | 101 | 7.8 |
| Government Employee | 80 | 6.2 |
| Industry Employee | 49 | 3.8 |
| Undergraduate Student | 48 | 3.7 |
| Management/Administrator | 20 | 1.5 |
| Retired/Not Employed | 18 | 1.4 |
| OS Respondents Use in Data Carpentry Workshops | n | % |
|---|---|---|
| Windows | 661 | 53.3 |
| Apple/Mac OS | 512 | 41.3 |
| UNIX/Linux | 50 | 4.0 |
| Not sure | 17 | 1.4 |
| Software Carpentry 1st Time Learners | n | % |
|---|---|---|
| Yes | 11525 | 94 |
| No | 697 | 6 |
| Data Carpentry’s U.S. Respondents’ Gender Identity | n | % |
|---|---|---|
| Female | 322 | 58 |
| Male | 223 | 40 |
| Transgender female | 2 | 0 |
| Prefer not to answer | 8 | 1 |
| Software Carpentry’s U.S. Respondents’ Gender Identity | n | % |
|---|---|---|
| Female | 322 | 58 |
| Male | 223 | 40 |
| Transgender female | 2 | 0 |
| Prefer not to answer | 8 | 1 |
| Data Carpentry’s U.S. Respondents Racial/Ethnic Identity | n | % |
|---|---|---|
| White | 316 | 54.0 |
| Asian | 152 | 26.0 |
| Hispanic or Latino(a) | 57 | 9.7 |
| I prefer not to say. | 28 | 4.8 |
| Black or African American | 25 | 4.3 |
| American Indian or Alaska Native | 4 | 0.7 |
| Native Hawaiian or Other Pacific Islander | 3 | 0.5 |
| Software Carpentry’s U.S. Respondents’ Racial/Ethnic Identity | n | % |
|---|---|---|
| American Indian or Alaskan Native | 16 | 0 |
| Asian / Pacific Islander | 751 | 23 |
| Black or African American | 144 | 4 |
| Hispanic or Latino | 204 | 6 |
| Native Hawaiian or Other Pacific Islander | 3 | 0 |
| White / Caucasian | 1916 | 58 |
| Prefer not to say | 197 | 6 |
| Multiple ethnicity / Other (please specify) | 95 | 3 |